A User-Centered Evaluation of a COVID-19 Intelligent Query System (COVID-IQS).

As the fight against COVID-19 continues, it is critical to discover and accumulate knowledge in scientific literature to combat the pandemic. In this work, we shared the experience in developing an intelligent query system on COVID-19 literature. We conducted a user-centered evaluation with 12 researchers in our institution and identified usability issues in four categories: distinct user needs, functionality errors, suboptimal information display, and implementation errors. Furthermore, we shared two lessons for building such a COVID-19 literature search engine. We will deploy the system and continue refining it through multiple phases of evaluation to aid in redesigning the system to accommodate different user roles as well as enhancing repository features to support collaborative information seeking. The successful implementation of the COVID-IQS can support knowledge discovery and hypothesis generation in our institution and can be shared with other institutions to make a broader impact.

Features Constituting Actionable COVID-19 Dashboards: Descriptive Assessment and Expert Appraisal of 158 Public Web-Based COVID-19 Dashboards.

BACKGROUND: Since the outbreak of COVID-19, the development of dashboards as dynamic, visual tools for communicating COVID-19 data has surged worldwide. Dashboards can inform decision-making and support behavior change. To do so, they must be actionable. The features that constitute an actionable dashboard in the context of the COVID-19 pandemic have not been rigorously assessed. OBJECTIVE: The aim of this study is to explore the characteristics of public web-based COVID-19 dashboards by assessing their purpose and users ("why"), content and data ("what"), and analyses and displays ("how" they communicate COVID-19 data), and ultimately to appraise the common features of highly actionable dashboards. METHODS: We conducted a descriptive assessment and scoring using nominal group technique with an international panel of experts (n=17) on a global sample of COVID-19 dashboards in July 2020. The sequence of steps included multimethod sampling of dashboards; development and piloting of an assessment tool; data extraction and an initial round of actionability scoring; a workshop based on a preliminary analysis of the results; and reconsideration of actionability scores followed by joint determination of common features of highly actionable dashboards. We used descriptive statistics and thematic analysis to explore the findings by research question. RESULTS: A total of 158 dashboards from 53 countries were assessed. Dashboards were predominately developed by government authorities (100/158, 63.0%) and were national (93/158, 58.9%) in scope. We found that only 20 of the 158 dashboards (12.7%) stated both their primary purpose and intended audience. Nearly all dashboards reported epidemiological indicators (155/158, 98.1%), followed by health system management indicators (85/158, 53.8%), whereas indicators on social and economic impact and behavioral insights were the least reported (7/158, 4.4% and 2/158, 1.3%, respectively). Approximately a quarter of the dashboards (39/158, 24.7%) did not report their data sources. The dashboards predominately reported time trends and disaggregated data by two geographic levels and by age and sex. The dashboards used an average of 2.2 types of displays (SD 0.86); these were mostly graphs and maps, followed by tables. To support data interpretation, color-coding was common (93/158, 89.4%), although only one-fifth of the dashboards (31/158, 19.6%) included text explaining the quality and meaning of the data. In total, 20/158 dashboards (12.7%) were appraised as highly actionable, and seven common features were identified between them. Actionable COVID-19 dashboards (1) know their audience and information needs; (2) manage the type, volume, and flow of displayed information; (3) report data sources and methods clearly; (4) link time trends to policy decisions; (5) provide data that are "close to home"; (6) break down the population into relevant subgroups; and (7) use storytelling and visual cues. CONCLUSIONS: COVID-19 dashboards are diverse in the why, what, and how by which they communicate insights on the pandemic and support data-driven decision-making. To leverage their full potential, dashboard developers should consider adopting the seven actionability features identified.

Development, Implementation, and User Evaluation of COVID-19 Dashboard in a Third-Level Hospital in Iran.

INTRODUCTION: The implementation of a dashboard enables managers to make informed and evidence-based decisions through data visualization and graphical presentation of information. This study aimed to design and implement a COVID-19 management dashboard in a third-level hospital in Mashhad, Iran. MATERIALS AND METHODS: This descriptive developmental applied study was conducted in the second half of 2020 in three stages, using user-centered design methodology in four phases: (1) specification of the application context, (2) specification of requirements, (3) creation of design solutions, and (4) evaluation of designs. Data collection in each phase was performed through holding group discussions with the main users, nominal group techniques, interviews, and questioners. The dashboard prototype for the data display was designed using the Power BI Desktop software. Subsequently, users' comments were obtained using the focus group method and included in the dashboard. RESULTS: In total, 25 indicators related to input, process, and output areas were identified based on the findings of the first stage. Moreover, eight items were introduced by participants as dashboard requirements. The dashboard was developed based on users' feedback and suggestions, such as the use of colors, reception of periodic and specific reports based on key performance indicators, and rearrangement of the components visible on the page. The result of the user satisfaction survey indicated their satisfaction with the developed dashboard. CONCLUSION: The selection of proper criteria for the implementation of an effective dashboard is critical for the health care organization since they are designed with a high-tech and content-based environment. The dashboard in the present study was a successful combination of clinical and managerial indicators. Future studies should focus on the design and development of dashboards, as well as benchmarking by using data from several hospitals.

Designing COVID-19 mortality predictions to advance clinical outcomes: Evidence from the Department of Veterans Affairs.

Using administrative data on all Veterans who enter Department of Veterans Affairs (VA) medical centres throughout the USA, this paper uses artificial intelligence (AI) to predict mortality rates for patients with COVID-19 between March and August 2020. First, using comprehensive data on over 10 000 Veterans' medical history, demographics and lab results, we estimate five AI models. Our XGBoost model performs the best, producing an area under the receive operator characteristics curve (AUROC) and area under the precision-recall curve of 0.87 and 0.41, respectively. We show how focusing on the performance of the AUROC alone can lead to unreliable models. Second, through a unique collaboration with the Washington D.C. VA medical centre, we develop a dashboard that incorporates these risk factors and the contributing sources of risk, which we deploy across local VA medical centres throughout the country. Our results provide a concrete example of how AI recommendations can be made explainable and practical for clinicians and their interactions with patients.

A Synergetic R-Shiny Portal for Modeling and Tracking of COVID-19 Data.

The purpose of this paper is to introduce a useful online interactive dashboard (https://mahdisalehi.shinyapps.io/Covid19Dashboard/) that visualize and follow confirmed cases of COVID-19 in real-time. The dashboard was made publicly available on 6 April 2020 to illustrate the counts of confirmed cases, deaths, and recoveries of COVID-19 at the level of country or continent. This dashboard is intended as a user-friendly dashboard for researchers as well as the general public to track the COVID-19 pandemic, and is generated from trusted data sources and built in open-source R software (Shiny in particular); ensuring a high sense of transparency and reproducibility. The R Shiny framework serves as a platform for visualization and analysis of the data, as well as an advance to capitalize on existing data curation to support and enable open science. Coded analysis here includes logistic and Gompertz growth models, as two mathematical tools for predicting the future of the COVID-19 pandemic, as well as the Moran's index metric, which gives a spatial perspective via heat maps that may assist in the identification of latent responses and behavioral patterns. This analysis provides real-time statistical application aiming to make sense to academic- and public consumers of the large amount of data that is being accumulated due to the COVID-19 pandemic.

Advantages of visualisations to evaluate and communicate adverse event information in randomised controlled trials.

BACKGROUND: Randomised controlled trials (RCTs) provide valuable information and inform the development of harm profiles of new treatments. Harms are typically assessed through the collection of adverse events (AEs). Despite AEs being routine outcomes collected in trials, analysis and reporting of AEs in journal articles are continually shown to be suboptimal. One key challenge is the large volume of AEs, which can make evaluation and communication problematic. Prominent practice is to report frequency tables of AEs by arm. Visual displays offer an effective solution to assess and communicate complex information; however, they are rarely used and there is a lack of practical guidance on what and how to visually display complex AE data. METHODS: In this article, we demonstrate the use of two plots identified to be beneficial for wide use in RCTs, since both can display multiple AEs and are suitable to display point estimates for binary, count, or time-to-event AE data: the volcano and dot plots. We compare and contrast the use of data visualisations against traditional frequency table reporting, using published AE information in two placebo-controlled trials, of remdesivir for COVID-19 and GDNF for Parkinson disease. We introduce statistical programmes for implementation in Stata. RESULTS/CASE STUDY: Visualisations of AEs in the COVID-19 trial communicated a risk profile for remdesivir which differed from the main message in the published authors' conclusion. In the Parkinson's disease trial of GDNF, the visualisation provided immediate communication of harm signals, which had otherwise been contained within lengthy descriptive text and tables. Asymmetry in the volcano plot helped flag extreme events that were less obvious from review of the frequency table and dot plot. The dot plot allowed a more comprehensive representation by means of a more detailed summary. CONCLUSIONS: Visualisations can better support investigators to assimilate large volumes of data and enable improved informal between-arm comparisons compared to tables. We endorse increased uptake for use in trial publications. Care in construction of visual displays needs to be taken as there can be potential to overemphasise treatment effects in some circumstances.

Rapid development of visualization dashboards to enhance situation awareness of COVID-19 telehealth initiatives at a multihospital healthcare system.

The COVID-19 pandemic has led to the rapid expansion of telehealth services as healthcare organizations aim to mitigate community transmission while providing safe patient care. As technology adoption rapidly increases, operational telehealth teams must maintain awareness of critical information, such as patient volumes and wait times, patient and provider experience, and telehealth platform performance. Using a model of situation awareness as a conceptual foundation and a user-centered design approach we describe our process for rapidly developing and disseminating dashboard visualizations to support telehealth operations. We used a 5-step process to gain domain knowledge, identify user needs, identify data sources, design and develop visualizations, and iteratively refine these visualizations. Through this process we identified 3 distinct stakeholder groups and designed and developed visualization dashboards to meet their needs. Feedback from users demonstrated the dashboard's support situation awareness and informed important operational decisions. Lessons learned are shared to provide other organizations with insights from our process.

Web-Based Video Conferences.

The recent shutting down of in-person events owing to the coronavirus disease 2019 (COVID-19) pandemic has elevated the popularity of web-based conferencing. This development provides an opportunity for educators to test their teaching skills on what, for many, is a new platform. Many of the basic elements of what constitutes an effective presentation are the same regardless of whether they are delivered in person or online. However, there are advantages and disadvantages of each mode of presentation, and understanding how to best leverage the features of an online platform will lead to a better educational experience for the presenter and audience. The effectiveness of any presentation is dependent on the ability of the speaker to communicate with the audience. This is accomplished by including as much audience participation as possible. Many of the techniques used to encourage audience participation in person can be adapted for use in online presentations (eg, the use of features such as chat, hand raising, polling, and question-and-answer sessions). In any type of presentation, both the quality of the content and the oral delivery are important. The author reviews the common elements of an effective presentation and how they can be optimized for online platforms. ©RSNA, 2020.

Integrative illustration for coronavirus outreach.

Two illustrations integrate current knowledge about severe acute respiratory syndrome (SARS) coronaviruses and their life cycle. They have been widely used in education and outreach through free distribution as part of a coronavirus-related resource at Protein Data Bank (PDB)-101, the education portal of the RCSB PDB. Scientific sources for creation of the illustrations and examples of dissemination and response are presented.

Geographic risk assessment of COVID-19 transmission using recent data: An observational study.

BACKGROUND: The US Centers for Disease Control and Prevention (CDC) regularly issues "travel health notices" that address disease outbreaks of novel coronavirus disease (COVID)-19 in destinations worldwide. The notices are classified into 3 levels based on the risk posed by the outbreak and what precautions should be in place to prevent spreading. What objectively observed criteria of these COVID-19 situations are required for classification and visualization? This study aimed to visualize the epidemic outbreak and the provisional case fatality rate (CFR) using the Rasch model and Bayes's theorem and developed an algorithm that classifies countries/regions into categories that are then shown on Google Maps. METHODS: We downloaded daily COVID-19 outbreak numbers for countries/regions from the GitHub website, which contains information on confirmed cases in more than 30 Chinese locations and other countries/regions. The Rasch model was used to estimate the epidemic outbreak for each country/region using data from recent days. All responses were transformed by using the logarithm function. The Bayes's base CFRs were computed for each region. The geographic risk of transmission of the COVID-19 epidemic was thus determined using both magnitudes (i.e., Rasch scores and CFRs) for each country. RESULTS: The top 7 countries were Iran, South Korea, Italy, Germany, Spain, China (Hubei), and France, with values of {4.53, 3.47, 3.18, 1.65, 1.34 1.13, 1.06} and {13.69%, 0.91%, 47.71%, 0.23%, 24.44%, 3.56%, and 16.22%} for the outbreak magnitudes and CFRs, respectively. The results were consistent with the US CDC travel advisories of warning level 3 in China, Iran, and most European countries and of level 2 in South Korea on March 16, 2020. CONCLUSION: We created an online algorithm that used the CFRs to display the geographic risks to understand COVID-19 transmission. The app was developed to display which countries had higher travel risks and aid with the understanding of the outbreak situation.